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ABSTRACT 

"Che   theory  of  test  data  selection  proposed  by  Goodenough  and  Gerhart  is 
examined  and  a  number  of  theoretical  and  pragmatic  deficiencies  are 
identified.  "Itie  concepts  of  a  revealing  test  criterion  and  a  revealing 
subdomain  are  proposed  to  overcome  some  of  these  weaknesses,  and  to  provide  a 
realistic  basis  for  a  theory  of  testing. 

A  subset  of  a  program's  input  domain  is  revealing  if  the  existence  of  one 
incorrectly  processed  input  implies  that  all  the  subset's  elements  are 
processed  incorrectly.  The   intent  of  this  notion  is  to  partition  the 
program's  domain  in  such  a  way  that  all  elements  of  an  equivalence  class  are 
either  processed  correctly  or  incorrectly.  A  program  test  set  is  then  formed 
by  choosing  one  element  from  each  class.  The  technique  is  especially  useful 
for  programs  v^ose  computations  depend  on  a  classification  of  their  input 
domain,  but  it  can  also  be  useful  for  programs  v^ich  process  their  entire 
domain  uniformly. 

A  methodology  for  forming  revealing  subdomain  partitions  is  described, 
and  illustrated  with  three  examples  to  v^ich  other  testing  methodologies  have 
been  applied  in  the  literature. 


for  uniform  testing  of  any  program  intended  to  meet  a  given  set  of 
specifications. 

In  Section  5  we  introduce  the  concepts  of  a  revealing  test  selection 
criterion  and  a  revealing  subdomain.  These  definitions  address  some  of  the 
deficiencies  in  the  Goodenough  and  Gerhart  theory,  and  are  intended  to  capture 
the  idea  of  testing  to  exhibit  the  presence  of  errors,  rather  than  to  be 
assured  of  their  absence.  Section  6  presents  a  methodology  for  constructing 
revealing  subdomains,  and  illustrates  its  application  to  several  example 
programs. 

2.  The  Goodenough  and  Gerhart  Definitions 

We  begin  by  examining  the  Goodenough  and  Gerhart  definitions  for  test 
criteria  properties  to  see  whether  they  have  captured  the  necessary  features. 
We  use  their  notation:  F  is  a  program,  D  the  input  domain,  and  R  the  output 
domain  for  F.  On  input  d  «  D,  F  (if  it  terminates)  produces  output  F{d)  ^  R. 
The   output  specification  for  F  is  given  by  OUT{x,y),  where  x  e  D  and  y  e  R.  F 
is  correct  on  input  d  (abbreviated  OK(d))  if  F(d)  exists  and  OUT(d,F(d) ) . 

A  test  T  for  program  F  is  simply  a  (finite)  subset  of  D.  A  test 
selection  criterion  C  is  a  predicate  on  subsets  of  D  which  selects  certain 
sets  of  inputs  as  appropriate  tests.  The  key  definitions  in  [5]  are  the 
following:  A  test  T  is  successful  iff  (Vt  e  T)(OK(t)).  A  test  selection 
criterion  C  is  reliable  iff  either  every  test  selected  by  C  is  successful,  or 
no  test  selected  is  successful.  C  is  valid  iff  whenever  program  F  is 
incorrect,  then  C  selects  at  least  one  test  set  T  which  is  not  successful  for 
F.  From  tliese  definitions,  the  fundamental  theorem  of  [5]  follows:  If  C  is  a 
reliable  and  valid  criterion,  then  any  test  selected  by  C  is  an  ideal  test. 
We  shall  call  a  criterion  which  is  both  reliable  and  valid  for  a  program  F,  an 


ideal  criterion. 

There  are  several  serious  deficiencies  in  the  theory  founded  on  this 
theorem.  First  is  that  the  concepts  of  reliability  and  validity  are  defined 
with  respect  to  the  entire  input  domain  of  a  program,  and  thus  allow  no 
discrimination  among  errors.  A  criterion  which  exposes  only  a  single  error  is 
valid  regardless  of  the  number  of  errors  present  in  the  program  in  question. 
We  discuss  a  way  to  avoid  this  problem  in  Section  5. 

The   second  difficulty  comes  about  because  all  the  above  definitions  are 
relative  to  a  single  program.  A  criterion  which  is  reliable  or  valid  for  r  is 
not  necessarily  so  for  F'.  The  problems  arising  with  tests  based  solely  on 
the  structure  of  a  program  are  avoided  in  [5] ,  but  the  tests  are  nevertheless 
based  entirely  on  the  behavior  of  the  single  program  under  consideration. 

Another  problem  is  the  lack  of  independence  of  the  properties  of  validity 
and  re?:iability.  Ttiis  problem  will  be  discussed  in  Section  3,  v^iere  we  show 
how  it  adversely  affects  the  theory's  practical  utility. 

Fourth,  neither  validity  nor  reliability  is  preserved  throughout  the 
debugging  process.  That  is,  a  criterion  which  is  valid  at  the  start  of 
debugging  does  not  necessarily  remain  so  for  every  intermediate  program 
produced  by  successive  corrections  made  on  the  way  to  an  error-free  program. 
Furthennore,  an  invalid  criterion  may  become  valid  as  the  program  is 
transformed.  The  same  is  true  for  reliability. 

From  the  point  of  view  of  the  programmer  who  has  just  designed  a  program 
or  a  person  assigned  the  task  of  testing  and  debugging  someone  else's  program, 
it  is  practically  impossible  to  find  test  selection  criteria  which  are  both 
reliable  and  valid,  since  these  properties  depend  on  the  nature  of  errors 
present  in  the  program.  The  situation  is  illustrated  with  several  simple 
examples. 


First,  if  progremi  F  is  correct,  i.e.,  if  (Vd  t  D)(OK(d)),  then  any  test 
will  be  successful  and  every  selection  criterion  C  is  reliable  and  valid.  Of 
course,  F  is  not  knov/n  to  be  correct,  and  the  programmer  must  use  other  means 
to  show  the  reliability  and  validity  of  C.  Such  other  means  will  be  the 
equivalent  of  a  proof  of  the  program's  correctness. 

On  the  other  hand,  if  F  is  not  correct,  there  is  no  way  of  knowing 
v*iether  a  criterion  is  ideal  without  knowing  the  errors  in  F.  The  first 
example  in  [5]  illustrates  this  point.  The  program  F  computes  d*d,  for  d  an 
integer,  wliile  the  output  specification  is  F(d)  =  d+d.  Since  F  is  correct  for 
d  =  0  and  d  =  2,  and  incorrect  for  all  other  inputs,  a  criterion  v^ich  selects 
as  tests  only  subsets  of  {0,2}  is  reliable  but  not  valid,  since  it  does  not 
indicate  the  error  in  F.  A  criterion  which  selects  subsets  of  {0,1,2,3,4} 
exposes  the  error  in  F,  and  is  therefore  valid,  but  is  not  reliable  since  T  = 
{0,2}  is  a  successful  test,  vrfiile  T  =  {0,1}  is  not  successful. 

A  slight  change  in  the  program,  while  retaining  the  same  output 
specification,  completely  changes  the  reliability  and  validity  of  these 
criteria.  If  F'  computes  d+2,  then  d  =  2  is  the  only  input  for  v^ich  a 
correct  answer  is  produced.  Thus,  choosing  subsets  of  {0,2}  becomes  a  valid, 
but  not  reliable  criterion  for  F' .  If  F' '  computes  d+5,  the  criterion  which 
selects  subsets  of  {0,1,2,3,4}  is  now  both  reliable  and  valid. 

The  problem  facing  the  tester  is  to  find  an  ideal  criterion  for  his 
program.  However,  as  the  above  examples  show,  there  is  no  way  to  know  if  a 
criterion  is  ideal  without  already  knowing  the  errors  in  the  program,  surely  a 
paradoxical  situation. 

3.  Practical  Testing  Considerations 

We  now  examine  the  usefulness  of  an  ideal  test  for  the  practical  problems 


of  program  writing  and  debugging.  Typically,  a  program  goes  through  many 
different  versions,  steps  of  refinement,  improvements,  modifications,  etc.,  as 
it  is  being  written.  It  is  desirable  to  verify  the  correctness  of  each  stage 
in  order  to  increase  confidence  in  the  correctness  of  the  next  one.  An  ideal 
test  criterion  for  one  version  is  not  necessarily  an  ideal  criterion  for 
another.  The  debugging  process  constantly  changes  the  program  being  worked 
on;  errors  are  located  and  removed,  and  sometimes  new  errors  are 
inadvertently  introduced. 

Consider  the  following  modification  of  the  example  presented  earlier: 
F(d)  =  (d*d)  +  3 
OK(d)  =  [(d+d)  +  6  =  F(d)] 

That  is,  F  is  supposed  to  double  the  integer  d  and  add  6,  but  instead  F 
squares  d  and  adds  3. 

Note  that  0K(3)  and  OK(-l),  but  ~OK{d)  for  all  other  d.  Tlius, 
C(T)  =  (T  =  {t}  and  t  e  {0,1,2})  is  a  valid  and  reliable  criterion.  Now 
suppose  that  by  running  F  on  one  of  the  input  values  of  T,  the  error  of  adding 
the  wrong  constant  is  located.  This  error  is  corrected,  and  now  F(d)  = 
(d*d)  +  6.  At  this  point,  OK(0)  and  0K(2),  but  ~OK(d)  for  all  other  d.  Hence 
C  is  no  longer  a  reliable  criterion,  since  some  tests  satisfying  C  will  detect 
the  remaining  error,  and  others  will  not. 

Note  that  the  reverse  situation  can  also  occur.  Thus,  for  example, 
C'(T)  =  (T  =  {t}  and  t  e  {-1,1,3})  is  valid  but  not  reliable  for  the  original 
program  containing  two  errors.  For  the  partially  corrected  program,  however, 
it  is  both  valid  and  reliable. 

Thus  properties  of  criteria  are  not  maintained  throughout  the  debugging 
phase,  nor  are  they  even  "monotonic"  in  the  sense  of  being  either  uniformly 
gained  or  preserved,  or  uniformly  lost  or  preserved. 


Are  reliabilty  and  validity  properties  which  testers  should  attempt  to 
give  their  tests?  The  authors  of  [5]  point  out  the  unreliability  of  test 
selection  criteria  based  solely  on  internal  program  structure  and  state  that 
the  success  of  tests  based  on  an  unreliable  criterion  is  poor  evidence  that  a 
program  contains  no  errors. 

In  practice,  however,  it  seems  extremely  unlikely  that  any  reasonable 
test  selection  criterion  would  be  reliable  in  the  Goodenough  and  Gerhart 
sense.  Typical  tests  or  groups  of  tests  are  constructed  to  check  the 
possibility  of  various  types  of  errors  which  may  exist.  The  criterion  which 
selected  these  tests  would  be  reliable  only  if  the  program  contained  either  no 
errors,  or  all  the  possible  errors.   (An  additional  possibility  is  that  the 
criterion  is  so  poorly  designed  that  it  exposes  none  of  the  errors  v^ich  do 
exist.)  Obviously  most  programs  will  be  somewhere  in  between  the  first  two 
cases,  and  we  should  not  expect  all  test  results  to  be  the  same.  The  third 
case  would  arise  only  if  the  criterion  were  invalid,  clearly  the  worst  type  of 
test. 

It  is  interesting  to  note  that  the  Goodenough  and  Gerhart  notions  of 
reliability  and  validity  are  not  independent;  at  least  one  of  the  two 
properties  must  hold  for  any  criterion. 

Theorem  ]  :   (VC)  (Val(C)  v  Rel  (C) ) 

This  is  true  because  an  invalid  test  criterion  exposes  no  errors,  and 
therefore  all  its  tests  are  successful.  Thus,  possession  of  either 
reliability  or  validity  alone  says  nothing  about  the  quality  of  a  test 
selection  criterion.  In  fact,  a  way  to  guarantee  validity  is  to  use  an 
unreliable  criterion,  and  as  we  saw  above,  most  criteria  probably  are 


unreliable. 

4.  Uniformly  Valid  and  Reliable  Test  Criteria 

We  have  seen  that  Goodenough  and  Gerhart's  properties  for  ideal  tests 
suffer  from  a  fault  similar  to  program  structure-based  tests,  namely, 
dependence  on  the  particular  program  being  tested.  A  possible  remedy  for  the 
situation  is  to  look  for  tests  whose  validity  and  reliability  are  dependent 
only  on  the  desired  output  specification.  Such  a  test  would  be  universal  for 
any  program  written  to  satisfy  the  given  specification.  Note  that  in  the 
definitions  of  validity  and  reliability  in  [5]  there  is  an  implicit  free 
variable  F,  referring  to  the  particular  program  under  consideration.  The 
formal  definitions  of  uniform  validity  and  reliability  given  below  bind  this  F 
with  an  outermost  universal  quantifier.  Note  that  we  have  also  modified  the 
notation  of  [5]  for  SUCC  and  OK,  by  including  the  program  F  as  an  explicit 
parameter. 

Given  an  input  domain  D  and  output  specification  OK(F,d),  criterion  C  is 
uniformly  valid  iff 

(VF)[(3  de  D)  (~OK(F,d))  =  (aTcD)(C(T)A  ~SUCC{F,T) )]  . 
Criterion  C  is  uniformly  reliable  iff 

(VF)(VT^,  T2£D)  [(C(Tj^)  a  C(T2))  =  (SUCC(F,T^)=  SUCC(F,T2) )  ]  . 

A  uniformly  ideal  test  selection  criterion  for  a  given  output 
specification  is  both  uniformly  valid  and  reliable.  Such  a  definition  would 
solve  all  the  program  dependent  difficulties  inherent  in  the  definitions  of 
[5].  Unfortunately,  however,  the  concept  of  a  uniformly  ideal  criterion  also 
has  serious  flaws.  Most  important,  for  any  significant  program  there  can  be 
no  uniformly  ideal  criterion  which  is  not  in  a  sense  trivial.  To  see  this, 
let  us  call  a  criterion  C  trivially  valid  if  the  union  of  all  tests  selected 


by  C  is  D.  Obviously  a  trivially  valid  criterion  is  valid.  However,  a 
criterion  C  which  is  not  trivially  valid  cannot  be  uniformly  valid  for  a  given 
output  specification,  since  for  any  element  d  not  included  in  any  of  C's 
tests,  a  program  can  be  written  which  is  incorrect  for  d,  and  correct  for 
D  -  {d}.  Ttius  we  have: 

Theorem  2:   A  criterion  C  is  uniformly  valid  if  and  only  if  C  is  trivially 
valid. 

A  similar  weakness  holds  for  uniform  reliability. 

Theorem  3:   A  criterion  C  is  uniformly  reliable  if  and  only  if  C  selects  a 
single  test. 

Proof:  If  C  selects  only  one  test  it  is  clearly  reliable  for  any  program. 
Suppose  C  selects  different  tests  T,  and  T^,  and  that  t  is  an  input  in  T,  ,  but 
not  in  Tp.  A  program  F  exists  which  is  corr'ect  on  all  the  inputs  in  To,  but 
incorrect  on  t.  Tl-ie  two  tests  thus  yield  different  results  for  F,  and  C  is 
not  reliable.   □ 

Let  us  nov;  consider  the  uniform  analogue  of  the  fundamental  theorem  of 
[5].  This  theorem  states  that  if  C  is  a  reliable  and  valid  criterion  for  a 
given  program  F,  and  if  F  runs  correctly  on  every  elonent  of  a  test  which 
satisfies  C,  then  F  runs  correctly  on  every  element  in  the  domain.  In 
contrast  to  this,  our  next  result  shoves  that  there  can  be  no  criterion  which 
uniformly  reveals  errors. 

Theorem  4:    (VC){\/T^  D)  [  C{T)  ^    (9  F)  (SUCC(F,T)  ^  ~SUCC(F,D))1 


Proof;  Since  there  is  some  input  datum  t  /  T,   let  F  be  a  program  which  is 
correct  on  T  and  v^ich  does  not  halt  on  t.   Q 

This  result  states  essentially  that  no  matter  wliat  criterion  is  used  to 
select  tests  for  a  given  specification,  and  no  matter  what  test  is  chosen  (if 
it  is  not  all  of  D) ,  there  will  always  be  a  program  v^iich  can  defeat  the  test. 
This  is  another  way  of  expressing  the  oft  quoted  statement  that  "testing  can 
only  reveal  the  presence  of  errors,  never  their  absence"  [3]. 

Other  similar  negative  results  have  been  proved.  Weyuker  [12]  has  shown 
that  there  can  be  no  algorithm  v^iich  can  decide  whether  or  not  a  given 
statement,  branch,  or  path  of  a  program  may  ever  be  exercised,  nor  whether  or 
not  every  such  unit  may  be  exercised.  Tlnus,  a  testing  methodology  which 
requires  the  generation  of  data  to  do  one  or  more  of  these  things  cannot  be 
guaranteed  to  terminate. 

Hovgden  [7]  showed  that  there  is  no  procedure  which,  given  an  arbitrary 
program  F  and  output  specification,  will  produce  a  non-empty  finite  test  set 
T  £  D  such  that  if  F  is  correct  on  T  then  F  is  correct  on  all  of  D.  The 
reason  behind  this  result  is  that  the  non-existent  procedure  is  expected  to 
work  for  all  programs,  and  we  thus  encounter  familiar  non-computability 
limitations.  Howden's  response  is  to  look  for  subclasses  of  programs  for 
v*iich  test-generating  procedures  can  be  successful. 

Similarly,  in  practical  terms,  one  does  not  have  to  design  test  criteria 
to  deal  with  every  possible  incorrect  program.  Programmers  do  not  write 
pathologically  wrong  programs,  but  rather  attempt  to  meet  the  desired  output 
specifications.  A  theory  of  testing  should  attempt  to  take  advantage  of 
classes  of  errors  which  are  known  to  occur  frequently,  and  should  capture  the 
essence  of  test  selection  criteria  which  can  expose  the  presence  of  such 
errors  or  guarantee  Lheii  absence. 
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5.  Revealing  Test  Selec±ion  Criteria 

The  theory  proposed  in  [5]  shows  that  testing  can  establish  the 
absence  of  errors  in  a  program,  if  the  test  selection  criteria  satisfy  the 
right  properties.  Both  the  theoretical  problons  and  the  difficulties  which 
arise  in  its  application,  however,  render  the  theory  of  little  pragrretic  use. 
As  we  saw  in  Sections  2  and  3,  finding  valid  and  reliable  criteria  for  real 
programs  is  all  but  impossible.  Thus,  although  theoretically  testing  can  be 
used  to  show  the  absence  of  errors,  pragmatically  it  rariains  true  that  testing 
can  only  expose  their  presence. 

It)  provide  an  effective  foundation  for  testing  real  programs,  a  theory 
of  testing  must  characterize  tests  which  uncover  errors.  In  addition,  a  single 
test  or  test  selection  criterion  must  not  be  expected  to  do  everything. 
Different  types  of  errors  are  exposed  by  different  test  methods;  different 
parts  of  a  program's  input  domain  may  be  subject  to  different  errors.  A  usable 
theory  of  testing  must  permit  a  test  set  to  contain  both  correctly  and  incorrectly 
processed  inputs. 

The  notions  of  revealing  test  criterion  and  revealing  subdomain  are 
intended  to  capture  these  properties  of  tests.  The  definitions  overccme  some, 
but  not  all  the  weaknesses  of  the  theory  in  [5].  In  the  following  section 
we  give  several  examples  of  programs  and  revealing  subdonains  constructed 
for  than.  In  practice,  it  is  important  to  use  infoniHtion  from  both  program- 
based  and  program- independent  sources.  The  approach  we  use  conbines  the 
techniques  of  path  dorrHin  testing  C7],  functional  testing  [9],  and  special 
values  testing  [6]. 

Formally,  a  test  criterion  C  is  revealing  for  a  siabset  S  of  the  input 
donain  if  whenever  S  contains  an  input  which  is  processed  incorrectly,  then 


n 


every  test  set  which  satisfies  C  is  unsuccessful.   Bquivalently,  if  any  test 
selected  by  C  is  successfully  executed,  then  every  input  in  S  produces  correct 
output.  A  formal  definition  is 

REVEALING  (C,S)  =  (3d  e  S)  (~OK(D))  ^  (VTcS)  (C(T)  ^  ~SUa:(T)) 
The  property  of  revealing  may  seon  like  a  very  strong  requirement  to  put 
on  a  test  selection  criterion,  but  the  domain  subset  parameter  allows  enough 
precision  to  make  the  definition  useful.  Any  test  selected  by  a  criterion 
which  is  revealing  for  subdcmain  S  is  an  ideal  test  for  S. 

Two  special  cases  of  revealing  criteria  are  worth  noting.  The  first 
is  when  the  subdomain  S  is  the  entire  input  dcmain  D  of  the  program. 
In  this  case  the  property  revealing  is  equivalent  to  the  conjunction  of 
validity  and  reliability,  so  long  as  vacuous  cases  cire  excluded. 

Theorem  5^:  A  non-empty  criterion  C  is  revealing  with  respect  to  the  domain  D 

of  a  program  if  and  only  if  C  is  both  valid  and  reliable. 

Proof:  Suppose  C  is  revealing.  If  there  are  any  errors  in  D  then  every  test 

meeting  C  must  be  unsuccessful.  C  is  therefore  reliable,  and  since  it  is 

non-empty,  it  is  also  valid. 

If  C  is  both  valid  and  reliable,  then  the  presence  of  any  error  guarantees 

ttat   every  test  meeting  C  will  be  unsuccessful. D 

CXir  characterization  of  a  good  test  criterion  is  thus  coextensive  with 
that  of  Goodenough  and  Gerhart,  although  as  noted  earlier,  the  attempt  to 
cover  all  possible  errors  with  a  single  test  criterion  is  unlikely  to  be 
successful . 
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The  second  special  case  occurs  wtien  the  criterion  allows  any  subset  of 
S  as  a  test  case.  The  two  parameters  then  become  one,  and  we  say  that  a  sub- 
danain  S  is  revealing  if  the  occurrence  of  an  error  is  S  inplies  the  failure 
of  any  subset  of  S.   In  other  words,  there  is  an  error  in  S  if  and  only  if 
all  inputs  in  S  produce  errors.  The  conjunction  of  these  two  special  cases 
inplies  that  eitlner  the  program  is  correct  or  the  program  processes  every  input 
incorrectly.  Clearly  tiiis  is  not  likely  to  occur  frequently,  nor  will  we  know 
when  it  does  occur.   It  is  not  of  pragmatic  interest. 

To  neke  use  of  a  property  as  strong  as  the  second  case  essentially 
requires  that  the  problem's  input  domain  be  partitioned  in  an  intelligent  and 
meaningful  way.  We  discuss  several  examples  below,  and  give  guidelines  for 
choosing  appropriate  subdcmains. 

If  we  have  a  revealing  criterion  for  a  subdomain  S  of  D,  running 
successful  tests  based  on  that  criterion  only  assures  us  of  the  correctness 
of  our  program  on  S.  Generally,  one  should  look  for  criteria  which  are 
revealing  on  subdcmains  whose  xmion  is  all  of  D.  Of  course,  if  a  particular 
problem  or  program  is  arrenable  to  testing  for  part  of  its  dcmain  and,  for 
example,  formal  verification  for  another  part,  then  both  methods  should  be 
used. 

In  a  study  of  s>-mbolic  testing  and  its  effectiveness  in  detecting 
errors  [8]  ,  Howden  states  that  "both  symbolic  testing  and  actual  data  testing 
are  unreliable  for  discovering  most  of  the  patli-dcmain  errors  in  the  sanple  pro- 
grams.... Actual  data  testing  can  reveal  the  error  if  tlie  prograrnner  is  lucky 
enough  to  choose  a  test  which  comus  froii  the  incorrect  part  of  the  danain." 

The  notion  of  a  revealing  subdarain  is  a  partial  solution  to  the  problem 
described  above.   If  a  program's  input  domain  can  be  successfully  divided  into 


1.   [81,  p.  274. 
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revealing  subdomains,  then  it  is  no  longer  necessary  to  rely  on  luck  in  choosing 
appropriate  tests.  Either  every  element  of  a  subdcniain  produces  the  correct 
output,  or  none  does. 
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6.  Constructing  Revealiny  Criteria 

Test  data  can  be  developed  from  both  the  program  under  consideration 
and  from  program- independent  sources  such  as  tlie  problem's  specifications,  the 
algorithm  used,  and  tlie  input/output  data  structures.   In  attenpting  to  find 
revealing  criteria  or  subdonains  for  a  program,  we  start  by  considering  the 
program  separately  from  the  program-independent  sources. 

The  input  dorein  is  first  partitioned  into  path  domains.  A  patli 
dorrain  consists  of  a  set  of  inputs  which  each  follow  the  same  path,  or  one  of 
a  family  of  related  paths,  through  the  program's  flew  graph.  A  second  pre- 
liminary partition,  the  problem  partition,  is  formed  on  the  basis  of  ccrnnon 
prcperties  implied  by  the  specifications,  algorithm,  and  data  structures. 
These  two  partitions  are  then  intersected  to  form  classes  which  are  the  basis 
for  test  selection. 

For  exartple,  a  problem  whose  single  input  is  an  integer  might  have 
specifications  which  suggest  a  partition  into  the  odd  and  even  integers. 
Analysis  of  the  progreim  graph  may  show  that  all  positive  inputs  follow  one  path, 
all  negative  inputs  another,  and  zero  is  processed  as  a  special  case.   Inter- 
secting the  problem  partition  with  tlie  path  dorain  partition  results  in  a  set 
of  five  classes:  odd  positive  integers,  odd  negative,  even  positive,  even 
negative,  and  zero.   If  our  understanding  of  the  problem  is  good,  these  classes 
are  likely  to  be  revealing  for  nany  errors. 

Usually  the  path  dorrain  partition  does  not  differ  so  markedly  from  tte 
problem  partition,  since  ultinately  the  prograra,  data  structures,  and  algorithm 
all  derive  fran  the  original  problem  specifications.  Tlie  differences  which  do 
exist  are  fruitful  places  to  loolc  for  errors.  This  is  the  basic  rationale 
bcliind  our  rrcthod  of  fonrdng  subdaiains. 
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Informally,  the  process  attenpts  to  cx)nstruct  a  partition  of  tlie  input 
dcirain  such  that  eleirents  in  an  equivalence  class  are  processed  the  saiie  way 
by  the  program,  and  in  addition  are  characterized  the  same  way  by  the  speci- 
fications and  the  algorithm.  The  classes  of  such  a  partition  are  usually 
revealing  for  most  errors. 

.  An  error  for  wiiich  not  every  subdcmain  is  revealing  is  usually  the 
result  of  a  patlxalogical  special  case  in  the  ccrrputation.  Frequently,  most 
of  the  subdonains  are  revealing  for  such  errors,  ttote  that  to  detect  an  error 
by  testing,  it  is  only  necessary  that  one  subdorain  be  revealing  and  produce 
incorrect  output  due  to  that  error.  Examples  of  this  type  of  error  are  given  for 
the  ejqxjnentiation  program  (Exanple  2) . 

Vfe  new  give  several  exairples  of  programs  and  revealing  subdoirain  con- 
struction. The  actual  test  data  consists  sinply  of  an  arbitrary  element  from 
each  subdorain. 

Elxample  1:  This  is  a  triangle  classification  problera  wluch  has  been  studied  by 
several  authors  [2,11].   The  problem's  specifications  are 

Input;  Three  positive  numbers  A,B,C;  with  A  s  B  >  C 

Output ;  Ihe  program  indicates  which  of  the  following  descrip- 
tions is  satisfied  by  A,  B,  and  C. 

1.  Ihey  cannot  be  the  sides  of  any  triangle 

2.  Ihey  are  the  sides  of  an  equilateral  triangle 

3.  They  are  the  sides  of  an  isosceles,  buL  not 
equilateral,  triangle 

4.  Ihey  are  the  sides  of  a  scalene  right  triangle 

5.  They  are  the  sides  of  a  scalene  obtuse  triangle 

6.  They  are  tlie  sides  of  a  scalene  acute  triangle 

Figiire  1  shows  tlie  program  presented  in  L2]  for  the  triangle  classification. 

Ihis  prc±)lem  is  especially  suitable  for  the  application  of  the  revealing 
subdonain  nethodology .  Since  the  very  purpose  of  the  program  is  to  classify 
its  input  domain,  there  is  an  oDvious  si)Ccification-based  i:)artition.   In  addi- 
tion, since  there  are  no  loops  in  tlie  program,  there  are  finitely  nany  path 


True 


Illegal 
Input 


A 

-(-    A 

*    A 

J 

B 

-e-    B 

*    B 

i 

C 

*■    C 

*    C 

i 

D 

-<-    B 

+    C 

Isosceles 
["riangle 


Equilatera] 
Triangle 


False 


Right 
^Triangle 


Obtuse 
^riangle 


\3; 


Acute 
Triangle 


Figure  1.   Flowchart  for  triangle  classification  problem 
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domains.  The  resulting  subdomains  (constructed  by  intersecting  the  problem 
partition  and  the  path  dorrain  partition)  therefore  separate  all  significant 
cases  for  this  problan,  and  we  can  expect  them  to  be  revealing  for  practically 
any  type  of  error. 

Each  path  domain  for  the  triangle  program  is  described  sinply  by  the 
conjunction  of  the  predicate  branches  taken  on  the  given  path.  Conditions  des- 
cribing the  six  path  domains,  P^  ~  ^f,'   sppear  in  Figure  2. 

To  form  the  specification-based  partition,  we  first  divide  the  universe 
of  triples  of  positive  numbers  into  legal  and  illegal  input  forms.  S.    is  all 
triples  such  that  ~ (A  >  B  >  C)  or  equivalently  (A  <  B)  v  (B  <  C) ;  these  are  the 
illegal  inputs.  For  the  legal  inputs  we  now  distinguish  two  conditions. 
S-  contains  all  triples  such  that  A  >  B  >  C  and  A  >  B-KT;  in  this  case  the  tri- 
angle inequality  is  not  satisfied,  and  A,B,C  cannot  be  the  sides  of  a  triangle 
(B  >  A+C  or  C  >  A+B  cannot  occur,  since  A  >  B  >  C) .  Inputs  such  that  A  >  B  >  C 
and  A  <  B+C  are  legally  ordered  and  represent  valid  triangles.  These  triples 
are  further  divided  into  six  subclasses: 


S,:  A  =  B  =  C  (equilateral) 

S.:  A  =  B  >  C  (isosceles) 

4 

S-:  A  >  B  =  C  and  A  <  B+C  (isosceles) 

o 

2         2  2 

S^;  A  >  B  >  C  and  A  =  B  +C  (right  scalene) 


'8' 


2    2  2 

S,,:    A  >  B  >  C  and  A  <  B  +C  (acute  scalene) 

2    2  2 

Sq:    a  >  B  >  C  and  A  >  B  +C   and  A  <  B+C       (obtuse  scalene) 


Intersecting  the  six  path  domains  with  the  eight  specification  donains 
results  in  the  nine  non-empty  subdomains  D.  -  D-  in  Figure  3.  These  are  the 
basis  for  the  test  data  used  for  the  triangle  program.  Figure  4  shows  a  test 


P  :  ~[  (A  >  B)  A  (B  >  C)  1   =   (A  <  B)  V  (B  <  C) 

P^:  (A  >  B  >  C)  A  (A^  =  B^+C^) 

P^:  (A  >  B  >  C)  A  (A^  >  B^+C^) 

P, :  (A  >  B  >  C)  A  (A^  <  B^+C^) 
4 

P  :  (A  >  B  >  C)  A  [  (A  =  B)  v  (B  -  C)  ]  a  ~[  (A  =  B)  a  (B  =  C)  ] 

E   (A  =  B  >  C)  V  (A  >  B  =  C) 

P,  :  (A  =  B  =  C) 

D 


Figure  2.    Path  domain  conditions  for  triangle 
classification  program 


D^    =    S^    n    P^:  (A  <  B)  V  (B  <  C) 


02=82"?:    (A  >  B+C)  A  (A  >  B  >  C) 


D^  =  S   n  P_:     (A  >  B+C)  A  (B  =  C) 


D.  =  S-  n  P^:     (A  =  B  =  C) 
4     3     6 


D-  =  S.  n  P^ :    (A  =  B  >  C) 
5     4     5 


Dg  =  S^  n  P^:    (A  >  B  =  C)  A  (A  <  B+C) 

D^  =  Sg  n  P  :    (A  >  B  >  C)  A  (A^  =  B^+C^) 

)8  =  S^  n  P^ 

)9  =  Sq  n  P3 


00=3^0?^:    (A  >  B  >  C)  A  (A^  <  B^+C^) 

Dq  =  Sq  n  P^:    (A  >  B  >  C)  A  (A^  >  B^+C  )  a  (A  <  B+C) 


Figure  3.    Subdomains  formed  by  intersection  of  problem 
partition  and  path  domain  partition 
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item  arbitrarily  cfhosen  from  each  subdomain,  together  with  the  program  output 
and  the  correct  output  for  each  item.  The   program  produces  an  incorrect 
result  for  subdcnains  D„  and  D  .   In  both  cases  the  error  is  that  the  program 
incorrectly  reports  a  triple  to  be  the  sides  of  a  valid  triangle  of  a  certain 
type.  D„  and  D^  are  revealing  for  these  errors;  all  their  eleiients  will  be 
processed  incorrectly  in  the  sane  way. 

Note  that  in  [2]   DeMillo,  Lipton,  and  Sayward  fail  to  detect  these 
errors;  one  of  their  test  cases  incorrectly  categorizes  (14,6,4)  as  an  obtuse 
triangle. 

While  our  treatment  of  the  triangle  prcblaii  produces  an  effective  set  of 
test  cases,  it  nevertheless  fails  to  deal  witli  a  type  of  error  wMch  may  be 
quite  cormon  in  real  programs.  Note  that  the  progran.  in  Figure  1  will  report 
that  (-3,-4,-5)  are  the  sides  of  an  acute  triangle,  however,  this  input  and  all 
other  triples  containing  negative  values  are  not  included  in  any  of  the  subdomains 
of  Figure  3.  Naturally,  we  developed  these  subdomains  on  the  assurtption  that 
the  inputs  would  all  be  positive  nunbers,  but  there  is  hardly  inore  reason  to 
make  this  assurtption  than  to  assume  the  inputs  will  be  in  non-decreasing  order. 
It  is  necessary  to  differentiate  between  two  types  of  invalid  input  situations. 
In  the  first  the  inputs  simply  are  of  tlie  wrong  form  for  the  given  problem,  as 
with  the  negative  triangle  inputs.  In  the  second  tlie  inputs  have  the  proper  form 
for  the  problem,  but  fail  to  satisfy  some  required  property,  such  as  the  triangle 
inequality. 

It)  ej<pose  the  effects  of  a  program's  failure  to  respond  properly  to  a 
wrong  form  type  of  input,  a  special  subdomain  can  be  established  for  all  such 
inputs.  The  second  type  of  invalid  input  should  be  treated  sinvLlarly  to  valid 
inputs,  since  in  general  some  processing  beyond  the  input  operations  is  needed 
to  decide  its  invalidity. 


Domain  Test  Data 

Dj  (1,  2,  3) 

D2  (14,6,  4) 

D3  (2,  1,  1) 

D4  (1,  1,  1) 

D5  (2,  2,  1) 

Dg  (3,  2,  2) 

D^  (5,  4,  3) 

Dg  (6,  5,  4) 

D„  (4,  3,  2) 


Correct  Output 
illegal  order 

not  a  triangle 

not  a  triangle 

equilateral 

isosceles 

isosceles 

right 

acute 

obtuse 


Actual  Output 
illegal  order 

obtuse 

isosceles 

equilateral 

isosceles 

isosceles 

right 

acute 


obtuse 


Figure  4.   Test  data  and  output  for  triangle  classification 
program 
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E^xanple  2:  This  program  is  sinple  enough  to  be  proved  correct,  and  many 
people  have  done  so. [1,10]  The  problem  is  to  calculate  x  to  the  power  y,  where 
X  is  an  integer  and  y  is  an  non-negative  integer.  Ihe  method  used  (Figure  5) 
does  the  ccnputation  in  time  proportional  to  log  y,  by  taking  advantage  of  the 
binary  representation  of  y. 

We  number  the  action  boxes  in  the  program's  graph,  and  see  that  the  set 
of  paths  through  it  is  a  subset  of  the  regular  set  1(3  v  23)*.  The  set  of 
feasible  paths  for  legal  inputs  is  1  v  1 (3  v  23)*23,  since  the  most  significant 
bit  in  the  binary  representation  of  y  must  be  a  1.  Note  that  the  path  taken  for 
input  (x  ,y  )  depends  only  on  the  value  of  y  . 

Since  tliere  are  infinitely  many  feasible  paths,  we  cannot  use  test  data 
frcni  every  path  donain.  We  choose  several  domains  which  are  representative  of 
different  types  of  y  values;  the  paths  amd  their  associated  y  values  are 
shewn  in  Figure  6. 

We  new  examine  the  problem  specif icaticxis  and  the  algorithm  to  find 
natural  ways  of  subdividing  the  x  and  y  darains.  An  obvious  partitioning  for 
x  is  based  on  whether  x  is  positive,  negative,  or  zero.  From  positive  x 
values  we  single  out  1,  since  it  is  easy  for  the  output  to  be  correct  if 
X  =^  1,  but  incorrect  otherwise.   We  similarly  single  out  x  =  -1.  The  x 
inputs  are  thus  partitioned  into  the  five  classes: 


\'' 

Xq   <     -1, 

X^: 

Xq  =     -1, 

X3: 

Xq=     0, 

^4  = 

Xq    =       1, 

X^: 

^c  ^    1- 

START 


i 


Input  (xQ,yQ) 


(x,y)  ^  (^o'V 
z  -^  1 


rue 


True 


z  -t-  z  *  X 


False 


y  ^  y  -  2 


X  -«-  X  "^  X 


STOP 


Figure  5.   Flowchart  for  fast  exponentiation  progr 


am 


Path  Domain    Description         Binary  form       Decimal  value 

of  path  of  Yq  of  Yq 


^1  1 

Y,         13^23,  n>l 


2 
•4 


Y,         1(23)",  n>l 


Y.         1(323)'^,  n>2 


Y  1(233)"23,  n>l 


10" 

2" 

1^ 

2"    -    1 

(10)" 

-f-(4"    -    1) 

(10)"l 

4-{4"+l    -    1) 

Figure  6.   Path  domains  of  exponentiation  program 
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The  specifications  and  algorithm  suggest  similar  natural  classes  for  ti\c 

e3^)onent  y.  Tlie  three  classes  y^  <  0,  y  =0,  and  y  >  0  should  be  considered, 

If 

as  well  as  certain  special  cases  when  y  is  positive.  The  case  where  y  =  2  is  of 

special  interest,  since  in  this  case  the  algorithm  works  by  accuniulating  the 

entire  paver  in  x,  and  does  not  use  z  until  the  last  loop  iteration.  Simi- 

larly  y  =  2  -  1  is  unusual,  since  the  power  is  accumulated  entirely  in  z. 

Note  that  tliese  two  si,)ecial  crses  for  y  ,  as  well  as  y  =0,  were  already 

identified  as  separate  classes  in  the  path  doi^xn  analysis  (See  Figure  6) . 

CJonbining  these  specification—  and  algorithm-based  classes  vrxtii   the 

path  dcmain  classes  gives  the  26  subdotains  sunmarized  in  Figure  7.  Since  Y 

represents  illegal  values  for  y,  we  can  form  a  single  subdomain  where  y  <  0 

and  X  is  arbitrary, 
o 

Ke  new  examine  scxte  errors  which  could  have  been  written  into  the  code, 

and  their  effect  on  the  program's  output;  our  dojective  is  to  see  for  which 

errors  the  subdonains  are  revealing. 

Error  1;  z  is  initialized  to  0  instead  of  to  1 

Effect;   Ihe  program's  output  is  always  0.  Ihis  is  incorrect  for  all.  inputs 
except  X  =  0,  y  >  0.  Thus  every  input  in  the  subdanains  built 
from  X,,  Xp,  X.,  and  X^.  will  produce  the  wrong  output;  inputs  in 
X,  x(Y   y   Y  ,  Y-)  produce  correct  output.  Tlie  output  is  incorrect 
for  the  single  member  of  X  x  Y^ .   Every  subdonain  is  revealing  for 
Error  1. 

Error  2:  Ihe  y  =  0  test  is  written  as  y  7^  0. 

Effect:   If  y  =  1,  the  program  loops.  If  y  7^  0,  the  output  is  1.  This  is 

incorrect  except  when  x  =  1,  or  x  =  -1  and  y  is  even.  All  subdaieins 

00         o 

are  revealing. 


X  input  classes  Y  input  classes 

X^:    Xq  <  -1  Y^:    y^  =  0 

Y3:    Yq  =  2"  -  1,  nil 

Y5:    Yq  =-1-{4"'^^  -  1),  nsl 


X^: 

^0 

=  -1 

X3: 

^0 

=    0 

X4: 

'^O 

=  1 

X3: 

^0 

>  1 

Subdomains:    X.  x  y.,   i  =  1,...,5;  j  =  1,...,5 
X  X  Y^ 


Figure  7.   Subdomains  for  exponentiation  program 
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Error  3;  The  test  for  odd  y  is  written  as  a  test  for  even  y. 

Effect;   Since  this  error  effectively  reverses  the  Yes  and  No  exits  of  the  test, 

w 
the  result  will  be  to  corpute  x  ,  where  w  is  the  I's  conplenent  of 

the  binciry  representation  of  y  .  The  program's  output  will  be 

X    ~  "^o  where  k  =  I  log  y  |  +1,  or  the  nurrber  of  bits  in  y  's 

binary  representation.  This  is  the  wrong  answer  unless  y  =0;  all  . 

svibdcrnains  are  revealing  for  tliis  error. 

Error  4:  Failure  to  test  for  y  odd;  all  loop  iterations  go  through  the 

z  -»-  z  *x  calculaticxi. 


Effect ;   Incorrect  result  produced  except  wl-ien  y  =0,  x  =0,  x  =1,  x  =-1 

o      o      o      o 

and  y  odd,  or  y  =2^-1.  Subdotvains  Y^  ^   X  ,  Y  x  all  X^,  Y  x  all  X. 

are  all  processed  correctly;  all  others  are  processed  incorrectly. 

All  subdomains  are  revealing. 

IWo  other  errors  for  which  every  subdcarain  is  revealing  are  E^-:  Writing  the 

z  ■*-  z  *  X  statement  as  z  ■«-  z  +  x,  and  E^ :  Writing  y  -<-  y  ^  2  as  y  -^  y-2 . 

o 

The  error  of  writing  x-»-x*  xcisx-^-x  +  xis  not  revealed  by  every 

one  of  the  26  subdomains.  For  inputs  (0,y  ),  (x  , 1),  and  for  the  pairs  (x  /y  )  = 

(2,2)  and  (2,3)  the  program  wotild  produce  the  correct  result.  Because  of  tlie 

last  three  cases  the  subdonains  (X,,  X^,  X.^,  X^)  x  Y^,  X^  x  Y^,  e^nd  Xr-   x  Y. 

1245     352      b3 

are  not  revealing.  All  the  other  subdomains  are  revealing,  however,  and  in  most 

of  these  the  wrong  answer  is  produced.  The  error  will  certainly  be  detected. 
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Example  3:  Tlie  program  in  Figure  8,  taken  from  Geller  [4],  is  supposed  to 
find  the  nunber  of  days  between  two  dates  in  the  same  calendar  year.  Ihe 
input  is  two  pairs  (nionthl,  dayl)  and  (iionth2,  day2) ,  and  a  year.  It  is 
assuned  that  the  first  date  precedes  the  second,  and  that  the  inputs  will 
always  be  valid  in  all  other  respect. 

Despite  two  proofs  of  correctness  in  [4],  the  program  as  it  appears  there 
has  both  s>Titactic  and  logical  errors.  We  have  corrected  the  syntactic  errors, 
but  left  the  logical  errors  to  denonstrate  the  use  of  revealing  subdomains. 
In  deriving  these  subdorains  we  emit  forming  explicitly  the  patli  domain  and 
problem  partitions. 

The  input  domain  divides  very  naturally,  based  on  the  structure  of  the 
input,  on  the  expected  output,  and  on  observations  about  tlie  program's  treat- 
ment of  inputs.  Ihe  subdcmains  are  as  follows: 
S, :  the  set  of  date  pairs  such  that  monthl  =  month2. 

Since  the  program  explicitly  tests  tor  tliis  conditicn,  it  is  clear  tliat  there 
is  significance  to  this  subdanain. 

For  the  remaining  subdanains  we  must  consider  dates  in  different  nonths; 

there  are  various  Vv'ays  to  make  tlie  divisions,  but  consideration  of  whether  or 

not  a  leap  year  calculation  is  involved  will  play  a  part  in  any  partitioning. 

S_:  monthl  >  2  and  mDnth2  >  monthl  +1 

Here  there  is  no  leap  year  calculation  and  at  least  one  intervening  nonth. 

S  :  monthl  ^  2   and  nonth2  ~   nonthl  +  1 

Again  there  is  no  leap  year  calculation,  but  also  no  intervening  montlis. 

S.:  monthl  =  1  and  month2  >  2 
4 

S^:  monthl  =  2  and  mDnth2  >  2 


procedure  calendar  (integer  value  dayl,  monthl,  day2,  month2,  year) ; 
begin 

integer  days; 

if  month2  =  monthl  then  days  :=  day2  -  dayl 

comment  if  the  dates  are  in  the  same  month,  we  can  compute 
the  number  of  days  between  them  immediately; 
else 
begin 

integer  array  daysin  (1:  12); 

daysin(l)  :=  31;  daysin{3)  :=  31;  daysin(4)  :=  30; 

daysin(5)  :=  31;  daysin{6)  :=  30;  daysin(7)  :=  31; 

daysin(8)  :=  31;  daysin(9)  :=  30;  daysin{10)  :=  31; 

daysin(ll)  :=  30;  daysin(12)  :=  31; 

if  ((year  rem  4)  =0)  or 

((year  rem  100)  =  0  and  (year  rem  400)  =  0) 
then  daysin(2)  :=  28 
else  daysin (2)  :=  29; 
comment  set  daysin (2)  according  to  whether  or  not 

year  is  a  leap  year; 
days  :=  day2  +  (daysin (monthl)  -  dayl); 
comment  this  gives  (the  correct  number  of  days  - 

days  in  complete  intervening  months); 
for  I  :=  monthl  +  1  until  month2  -  1  do 

days  :=  daysin(I)  +  days; 
comment  add  in  the  days  in  complete  intervening  months; 
end; 
write (days) 
end; 


Figure  8.   Calendar  computation  program 
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S  and  S  cxjntain  inputs  which  require  leap  year  calculations.  To 

account  for  the  different  ways  in  wMch  a  year  can  be  a  leap  year,  we  further 

divide  S  and  S^  into  four  subclasses: 
4      5 

Y^ :  year  rem  4=0  and  year  rem  100  ^  0 

Y  :  year  roii  100  -   0  and  year  roii  400  ^   0 

Y  :  year  rem  400  =  0 

Y  :  year  rem  4  7^  0 

Ihe  final  set  of  subdornains  is  thus  S  ,  S  ,  S  ,  S  n  Y  ,  and  S  n  Y.  ,  i  =  1,2,3,4 
Ihe  most  obvioijs  types  of  errors  for  the  calendar  program  are  "off -by-one" 
errors.  We  might  e>53ect  the  program's  output  to  be  the  correct  value  +1,  or  to 
be  off  by  cne  month's  worth  of  days. 

Note  that  all  the  subdcnains  are  pairwise  disjoint,  and  tliat  their  union 
is  the  problem's  entire  domain.  If  we  can  shew  that  each  subdcmain  is  revealing 
for  a  given  set  of  errors,  and  then  run  successful  tests  on  elements  fron  each 
cffie,  we  shall  have  showed  tJiat  the  program  does  not  contain  the  specified  errors. 

Since  the  result  is  to  be  the  difference'  between  two  dates,  the  correct 

result  for  elements  in  S  is  of  the  forra  day 2  -  dayl+k,  k  an  integer.  Since 

these  ejqjressions  define  a  set  of  parallel  lines,  tJie  correct  e>qjression  is 

ccnpletely  determined  by  a  single  input  pair  together  with  its  correct  output. 

totice  that  if  we  were  concerned  about  v^iriether  the  proper  expression  should  be 

dayl  -  day2  +  k  rather  tlian  day2  -  dayl  +  k,  then  S,  would  not  be  revealing. 

These  expressions  are  no  longer  nonintersecting,  since  when  dayl  =  day2, 

(dayl  -  day2  +  k)  =  (day2  -  dayl  +  k) .  Ihus  if  we  feel  that  incorrect  order 

of  subtraction  is  a  potential  error,  then  S^  must  be  further  divided  into  the 

two  subdornains:        ^   .  .,.      ...    -,-,,   ^     ^ 

S  '  :    monthl  -  nonth2  and  da^-l  =  day 2 

S  '  '  :   monthl  =  mDnth2  and  dayl  ^  day2 


23 


Ihe  two  new  subdonains  are  revealing  for  both  types  of  errors. 

Next  we  note  that  the  table  of  days  in  is  initialized  correctly  for  the 
nvcanths  1,  3,  4, . . .  ,12.  lliis  assures  that  the  subdonains  S  and  S  are  revealing. 
S  and  S  were  separated  since  the  program  does  additional  processing  when 
intervening  months  occur.  Typical  errors  for  an  S  input  are  to  add  one  too 
fev  or  one  too  many  intervening  months,  and  tliese  errors  would  occur  for  every 
S  input.  Similarly  for  S  ,  if  an  intervening  nonth's  days  were  incorrectly- 
added  in,  the  error  would  occur  for  all  of  S-. 

Ihe  subdonains  fron  S  and  S^.  reveal  the  errors  existing  in  tlie  original 
program.   Any  input  from  S  n  Y.  or  S  n  Y.,  i  =  1,3,4,  produces  an  incorrect 
number  of  days  for  February,  and  will  result  in  an  incorrect  output.  The  inputs 
in  S  n  Y  and  S  n  Y  all  produce  correct  resuJ.ts. 

Although  the  nost  likely  off-by-one  errors  will  be  revealed  by  all  the 
subdorains  constructed  in  the  exaitple,  certain  types  of  "double"  off-by-one 
errors  will  not  be  revealed.  Suppose  that  both  indices  on  the  for  loop  which 
adds  in  ccnplete  intervening  months  are  one  too  liigh,  i.e.,  the  statoiient  is 

for  I:  =  montlil  +  2  until  mDnth2  do 
Ihen  the  proper  number  of  months  will  be  added  to  the  sum  of  days,  but  they  will 
be  the  wrong  months.  If  the  inputs  are  nionthl  =  4,  month2  =  6  the  error  will 
be  detectcKi.  Mcaithl  =  4,  month2  =  7,  however,  will  produce  the  correct  output, 
for  the  wrong  reason. 

As  usual,  to  guarantee  detecting  this  error,  a  finer  input  partition  is 

necessary.  Because  of  the  calendar's  irregularity,  a  separate  class  is  needed 

II 

for  each  legal  pair  of  input  months.  There  are  ^    (12  -  i)  =66  such  pairs. 

A  correct  version  of  the  program  contains  the  following  replacement  for  the  if 

staternsnt  in  Figure  8. 

if  ((year  rcjn  4)  =  0  and  (year  rem  100)  /   0) 
or  ( (year  r^i  400)  =  0) 
then  d.-iysin(2)  :=  29 
else  daysin(2)  :-  28; 
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The  three  exaitples  in  this  seption  were  chosen  to  shew  the  applicability 
of  the  revealing  subdcmain  methodology  to  a  range  of  c3ifferent  types  of 
programs.  They  have  all  appeared  in  the  testing  literature,  and  have  had 
diverse  testing  nethodologies  applied  to  them. 

Our  technique  seons  most  applicable  to  programs  whose  purpose  is 
either  a  classification  of  their  inputs,  or  a  ccjrputation  based  on  such  a 
classification.  However,  it  can  also  be  effective  for  programs  which  process 
their  entire  dorein  unifontily. 

Since  the  goal  of  the  triangle  pr<±)lan  is  to  classify  its  inputs, 
it  is  obviously  well  suited  to  a  methodology  which  requires  a  dcnein  partition. 
Ihe  errors  detected  in  this  program  indicate  that  one  must  not  rely  solely 
an  the  partition  induced  by  the  program,  but  must  also  take  into  account 
the  requirements  of  the  prcblem. 

The  calendar   ccnputation,  although  not  directly  a  partitioning  of 
the  inputs,  is  nonetheless  based  on  a  classification  of  dates.  The  intersection 
of  the  problen  and  path  dotmin  partitions  which  was  derived  for  the  calendar 
problon  expresses  this  classification.  The  problem  is  just  as  well  suited 
to  the  revealing  si±)domain  technique  as  the  triangle  problem. 

The  exponentiation  problon  is  the  exaitple  to  which  the  methodology 
is  the  least  c±)viously  applicable;  its  conputation  does  not  make  use  of  any 
significant  partition  of  the  input  donain.  However,  a  partition  of  the  domain 
based  on  both  the  structure  of  the  data  and  the  program's  treatment  of  inputs 
was  sufficient  to  provide  a  set  of  test  cases  which  effectively  detected  many 
likely  errors. 
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7.  Surmary  and  Future  Work 

AlthDugh  program  testing  has  been  a  ccmron  practice  since  the  first 
days  of  progranming,  until  recently  there  has  been  almost  no  investigation 
into  the  theoretical  basis  of  testing.  The  first  efforts  to  construct  such  a 
theory  showed  the  inadequacy  of  tests  based  solely  on  program  structure,  and 
gave  conditions  under  which  a  successful  test  can  demonstrate  that  a  program 
is  error-free. 

However,  the  notion  of  an  ideal  test  criterion  is  both  program-  and  error- 
dependent,  and  thus  does  not  provide  a  usable  characterization  of  tests.  A 
program  independent  version  of  an  ideal  criterion  is  unfortu  tely  also  lonusable, 
since  no  realistic  criterion  can  possess  the  necessary  properties. 

The  notion  of  a  revealing  criterion  was  introduced  to  remedy  the  depend- 
ence between  validity  and  reliability,  and  to  allow  designing  tests  for  sub- 
domains  of  a  problem's  entire  donain.  A  theory  of  testing  should  establish 
realistic  and  useful  properties  of  good  tests  and  test  criteria.  Defining 
reliability  and  validity  in  terms  of  a  program's  entire  dorain  requires  a 
good  test  to  do  too  much.  Defining  good  test  criteria  in  terms  of  restricted 
subdomains  allows  each  criterion  to  concentrate  on  likely  local  errors,  and 
increases  one's  ability  to  find  good  tests. 

Much  work  remains  to  be  done  in  making  the  concepts  of  a  revealing 
criterion  and  subdomain  applicable  to  a  wide  class  of  problems  and  programs. 
One  of  the  most  important  tasks  is  to  find  more  systematic  or  formal  methods  of 
constructing  the  problem  partition.  This  will  not  be  easy,  since  finding  a  good 
problem  partition  is  quite  similar  to  the  task  of  creating  the  program  itself. 
Since  the  latter  is  so  difficult  to  formalize,  we  may  expect  the  former  to  be 
difficult  also. 
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On  the  tJieoretical  side,  we  may  ask  under  what  conditions  a  subdcmain  can 
be  revealing  for  every  possible  error.  Is  this  possible  for  any  subdomain 
other  than  the  trivial  case  of  a  singleton  set?  If  we  cannot  achieve  universally 
revealing,  non-trJ.vial  subdomains,  how  close  can  we  cone?  The  problon  is  to 
combine  realistically  attainable  characteristics  of  good  tests  with  a  reasonable 
level  of  confidence  in  the  test  results.  The  notion  of  revealing  brings  us  a 
little  closer  to  this  goal. 
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